A Fast Clustering System for a Huge Number of Nucleotide Sequences

Single pass sequences of mRNA, called ESTs, have been determined extensively. They have been accumulated in the dbEST database in GenBank. The number of ESTs in dbEST has become more than eight million in August 2002. By clustering and assembling ESTs, we can conduct the following analyses. First, we can obtain complete ORF sequences based on ESTs that are fragment sequences of mRNA and do not include complete open reading frames (ORFs). Second, we can examine expression profiles of genes by counting the number of ESTs obtained from each library in each cluster. Third, redundancy of EST sequences can be reduced by clustering. Finally, we can pick up ESTs that are representatives of clusters. Full-length sequences were determined for clones of representative ESTs chosen from clusters of unknown genes during the full-length human cDNA sequencing project supported by New Energy and Industrial Technology Developmental Organization (NEDO) in Japan [2]. To select clones for full-length sequencing, more than one million EST sequences were determined and clustered in this project. Now 21,243 human full-length cDNA sequences obtained in this project are published in DDBJ. Thus clustering and assembling of a huge number of ESTs are indispensable for EST analyses. It is computationally hard to see the whole relationship among millions of sequences for the purpose of clustering. Some currently available tools require more than a few days to complete clustering of such a huge number of sequences, while it is impossible for other tools to complete the task. In this paper, we present an algorithm for clustering and assembling that can complete the entire process for clustering and assembling in O(N log N) time where N is the number of sequences. We also implemented this algorithm and succeeded in assembling more than a million sequences within half a day.